Design, Implementation, and Performance of MPI on Portals 3.0

The emergence of cluster computing as a viable platform for high performance computing has been realized due to significant performance increases in commodity computing and networking hardware. In particular, relatively inexpensive programmable network interface cards (NICs), such as Myrinet (Boden et al., 1995), that are capable of delivering gigabit-per-second speeds, have allowed for much research into low-level message passing protocols and message passing interfaces (von Eicken et al., 1992, 1995; Ishikawa et al., 1996; Pakin et al., 1997; Myricom, Inc., 1997; Compaq et al., 1997; Prylli and Tourancheau, 1998). Most of this research has been focused on delivering latency and bandwidth performance as close as possible to the limitations of the hardware. In several aspects, the research on clusters of personal computers (PCs) with gigabit networking hardware is addressing many of the same problems that proprietary distributed-memory message passing parallel machines of the early 1990s faced. Despite the differences in hardware architecture between custom-built parallel machines and today’s PC cluster, many of the issues with respect to delivering network performance to parallel applications are similar. The Portals (Brightwell et al., 1999, 2002) data movement interface (Portals 3.0) is an evolution of networking technology initially developed for large-scale, distributed memory, massively parallel systems. Portals began as a key component of our lightweight compute node operating systems (Maccabe et al., 1994; Shuler et al., 1995), and has evolved into a functional interface that can be implemented efficiently for different operating systems and networking hardware. In particular, Portals provides the necessary building blocks for higher-level protocols to be implemented on programmable or intelligent network interfaces without providing mechanisms that are specific to each higher-level protocol. This paper describes how these building blocks and their associated semantics can be combined to support the protocols needed for a scalable, high performance implementation of the Message Passing Interface (MPI) Standard (MPI Forum 1994). Portals is the basis for the Computational Plant (Cplant) (Brightwell et al., 2000) cluster at Sandia National Laboratories, and the MPI implementation described in this paper has been used on our large-scale production machines for the last two years.